Storage remote copy system

ABSTRACT

In the conventional storage system, consistency is ensured only for writing from a single storage. Asynchronous remote copying and local replication are also alternately suspended in the conventional system, so that the suspension time increases and the volume in which consistency is obtained becomes old. Two local replicas are prepared, for a volume containing stored data that is transferred from the main site by asynchronous remote copying, in the storage at the sub-site. Each pair of local replicas is alternately suspended by a time-specific suspension command according to the time of the timestamp attached to the write data, and replica data in which the time sequence is secured is continually prepared. When a failure occurs at the main site, the data is recovered using the replica data.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.10/950,577, filed Sep. 28, 2004, which application also relates to andclaims priority from Japanese Patent Application No. 2004-231789, filedon Aug. 9, 2004, the entire disclosures of which are incorporated hereinby reference.

BACKGROUND OF THE INVENTION

The present invention relates to an information processing system inwhich a plurality of storage systems are connected by a network, and italso relates to a technique for data transfer and disaster recovery inan information processing system.

Temporary suspension of business, data loss, and the like can occur inan information processing system that contains a storage system when afailure occurs in the information processing system due to a powerfailure, a fire, an earthquake, or the like. In order to prevent suchsituations, the same type of information processing system is placed ina remote location which is not affected by fire, earthquake, or thelike, and duplicate information is prepared by transferring the datawritten in one information processing system (hereinafter referred to asthe main site) to the information processing system allocated at theremote location (hereinafter referred to as the sub-site). A techniqueexists for performing this transfer and duplication of data using anetwork (hereinafter referred to as remote copying) in order to obtainthese effects.

The term “remote copying” refers to the transfer of data stored at amain site from the main site to a sub-site. Backing up of data at aremote location, business continuity, and disaster recovery can therebybe performed.

Remote copying includes two types of methods: synchronous remote copyingand asynchronous remote copying. In synchronous remote copying, astorage system at the main site returns, to a computer (hereinafterreferred to as a host), a response to a write request from the hostafter data transfer to the sub-site is completed. There is, therefore,no data loss in synchronous remote copying, and the consistency of thedata is ensured. However, as the line delay between sites increases, anI/O delay occurs in the main site between the host and the storagesystem.

In asynchronous remote copying, the storage system at the main siteperforms data transfer to the sub-site after returning, to the host, aresponse to a write request from the host. A decrease in I/O performancebetween the host and the storage system is thereby less likely to occureven if there is a long distance between sites, but the possibility ofdata loss occurring increases in comparison to synchronous remotecopying, and the sequence of the data is not ensured.

Assurance of data consistency in asynchronous remote copying isdescribed in Japanese Laid-open Patent Application No. 2002-149499.Specifically, a method is disclosed in this publication wherebyadditional information is attached to the written data from the host,and a sorting of the data is performed at the remote system based on theadditional information to ensure consistency.

A technique called NanoCopy is also described in “The Hitachi NanoCopyAdvantage”, [online], June 1999, Hitachi Data Systems Corporation,Internet <URL: http://www.hds.com/pdf/wp134 ₁₃ nanocopy.pdf> as a methodfor ensuring the consistency of written data in asynchronous remotecopying across a plurality of storage systems. NanoCopy suspends theasynchronous remote copying of a plurality of storage systems at acertain time and creates a replica of the volume at a certain time. Byregularly repeating this operation, there continually exists a replicaas a volume having consistency at some future time.

A method is also disclosed in Japanese Laid-open Patent Application No.H7-72981 for acquiring a replication of the volume at a certain time athigh speed within the same storage. Volume replication by the methoddisclosed in this publication will be referred to hereinafter as asnapshot.

In the method disclosed in Japanese Laid-open Patent Application No.H7-72981, a volume used for saving data (hereinafter referred to as avolume pool) is secured in advance. The writing performed in thereplication source volume subsequent to the replication command is thenprocessed according to the steps described below.

(A) It is confirmed after the replication command whether the writingconstitutes the first update for the relevant data area. Step B isexecuted if the writing is the first, and step C is executed if thewriting is not the first.

(B) The contents prior to updating of the data area to be written to arecopied to the volume pool, the correspondence information of thereplication source area to the area of the volume pool targeted forcopying is stored, and step C is executed.

(C) The replication source volume is updated.

When data is read from the replication target after the replicationcommand, processing is performed according to the following steps.

(D) It is confirmed using the correspondence information whether thearea, for which there was a read request, has been copied to the volumepool; and, when it has been copied, step E is executed, and step F isexecuted when it has not been copied.

(E) The data prior to updating is returned from the volume pool usingthe correspondence information.

(F) The data is returned from the replication source volume.

Replication by a snapshot can create replication with a smaller volumecapacity than is achieved in volume replication by mirroring.

SUMMARY OF THE INVENTION

In the system of Japanese Laid-open Patent Application No. 2002-149499,consistency is ensured only for writing from a single storage unit.Also, asynchronous remote copying and local replication are alternatelysuspended in the system of this publication, so that the suspension timeincreases and the volume in which the consistency is obtained becomesold.

Therefore, an information processing system is disclosed herein wherebyasynchronous remote copying that ensures consistency among a pluralityof storage devices is performed without suspension of the asynchronousremote copying.

The information processing system has a first site connected to a firstcomputer, and it is provided with a first storage system and a secondstorage system; and a second site having a third storage system isconnected to a second computer and to the first storage system, while afourth storage system is connected to the second computer and to thesecond storage system. The third storage system has a first memory areafor storing data transferred from the first storage system, and a secondmemory area and third memory area for storing a copy of the data storedin the first memory area of the third storage system. The fourth storagesystem has a first memory area for storing data transferred from thefirst storage system, and a second memory area and third memory area forstoring a copy of the data stored in the first memory area of the fourthstorage system. The first storage system and the second storage systemeach receive data with an attached writing time from the first computer.The first storage system transfers, to the third storage system, thedata received from the first computer and a copy of the writing time.The second storage system transfers, to the fourth storage system, thedata received from the first computer and a copy of the writing time.The third storage system writes, to the first memory area in the thirdstorage system, the data received from the first storage system in thetime sequence in which the data was attached. A copy of the data writtenin the first memory area in the third storage system is written to thesecond memory area in the third storage system in the time sequence inwhich the data was attached. The fourth storage system writes, to thefirst memory area in the fourth storage system, the data received fromthe second storage system in the time sequence in which the data wasattached. A copy of the data written in the first memory area in thefourth storage system is written to the second memory area in the fourthstorage system in the time sequence in which the data was attached. Whencopying is completed to the second memory area in the third storagesystem regarding the data with an attached writing time that is prior toa first prescribed time specified by the first computer, the thirdstorage system suspends writing, to the second memory area in the thirdstorage system, of a copy of the data written in the first memory areain the third storage system. The third storage system then initiateswriting of a copy of the data written in the first memory area in thethird storage system to the third memory area in the third storagesystem. When copying is completed to the second memory area in thefourth storage system regarding the data with an attached writing timethat is prior to a first prescribed time specified by the firstcomputer, the fourth storage system suspends writing, to the secondmemory area in the fourth storage system, of a copy of the data writtenin the first memory area in the fourth storage system. The fourthstorage system then initiates writing of a copy of the data written inthe first memory area in the fourth storage system to the third memoryarea in the fourth storage system.

In an information processing system having a first site and a secondsite, the consistency of the data stored in a volume at the first siteand of the data stored in a volume at the second site at a certain timecan be ensured without suspending asynchronous remote copying from thefirst site to the second site.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram depicting an example of the informationprocessing system of the first embodiment;

FIG. 2 is a diagram depicting an example of the functional configurationof the storage systems contained at the main site;

FIG. 3 is a diagram depicting an example of the functional configurationof the storage systems contained at the sub-site;

FIG. 4 is a diagram depicting an example of the pair information;

FIG. 5 is a diagram depicting an example of the RC data information;

FIG. 6 is a diagram depicting an example of the LR managementinformation;

FIG. 7 is a diagram depicting an example of the RC pair information;

FIG. 8 is a flow diagram depicting an example of the processing flow ofthe operation of LR pairs LR1 and LR2 during normal operation;

FIG. 9 is a flow diagram depicting an example of the processing flowwhereby service is restarted at the sub-site after a failure occurs atthe main site;

FIG. 10 is a flow diagram depicting an example of the processing wherebythe main site is recovered and service is restarted at the main siteafter restarting service at the sub-site;

FIG. 11 is a flow diagram depicting an example of the processing forexecuting an At-Time-Suspend of the local replication function;

FIG. 12 is a flow diagram depicting an example of the LR volume electionoperation when a failure occurs;

FIG. 13 is a flow diagram depicting an example of the operation wherebyAt-Time-Suspend is performed for the LR pair;

FIG. 14 is a diagram depicting an example of the snapshot operation usedin the second embodiment;

FIG. 15 is a diagram depicting an example of a pair in the asynchronousremote copying and the local replication function;

FIG. 16 is a diagram depicting an example of the software configurationof the main site host;

FIG. 17 is a diagram depicting an example of the software configurationof the sub-site host;

FIG. 18 is a diagram depicting an example of the functionalconfiguration of the storage systems contained at the sub-site;

FIG. 19 is a flow diagram depicting an example of the operation wherebythe snapshot is created during normal operation;

FIG. 20 is a diagram depicting an example of the snapshot information;

FIG. 21 is a diagram depicting an example of the processing wherebyservice is restarted at the sub-site after a failure occurs in the mainsite;

FIG. 22 is a flow diagram depicting an example of the processing wherebythe main site is recovered and service is restarted at the main siteafter service is restarted at the sub-site; and

FIG. 23 is a flow diagram depicting an example of the At-Time-Snapshotoperation of the snapshot function.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

Embodiments of the present invention will next be described withreference to the drawings. However, it should be understood that thepresent invention is not limited by the present embodiments.

First Embodiment

FIG. 1 is a diagram depicting an example of an information processingsystem in which the present invention is applied.

The information processing system is composed of a main site 101 and asub-site 102 that are located at a distance from each other. The mainsite 101 is composed of a host 111, a plurality of storage systems 131,and a network 121 for connecting the host 111 with the plurality ofstorage systems 131. The sub-site 102 is composed of a host 112, aplurality of storage systems 132, and a network 121 for connecting thehost 112 with the plurality of storage systems 132.

The storage systems 131 are each composed of a processor 153; memory154; a host I/O interface 152 for receiving an I/O request from thehost; an asynchronous RC interface 151 connected to the network forperforming asynchronous remote copying from the storage systems 131 tothe storage systems 132; and a volume 141 for storing data. Data writtenfrom the host 111 is stored in the volume 141.

The storage systems 132 have volumes 142, 143, and 144 instead of thevolume 141 of the storage systems 131, and their configuration isotherwise the same as that of the storage systems 131. Data transferredfrom the storage systems 131 by asynchronous remote copying is stored inthe volume 142. Data copied at a certain time (hereinafter referred toas a replica) from the data stored in the volume 142 is stored in thevolumes 143 and 144. The function whereby this replica is created isreferred to hereinafter as the local replication function.

The local replication function is a function for creating a replica of avolume within the same storage system. The volume for storing thereplica created by the local replication function is also referred to asthe replication volume.

The volumes may also be composed of a plurality of physical disks.

The network 122 is a network for performing data transfer between thestorage systems 131 and the storage systems 132. The network 122 isconnected to the asynchronous RC interface 151 of the storage systems131 and storage systems 132.

The host 111 and the host 112 are computers on which an applicationprogram operates for performing transaction processing and other serviceusing the volume 141 of the storage systems 131 connected to the host111 or the volume 142 of the storage systems 132 connected to the host112. Usually, the host 111 performs service and the host 112 is astandby host for taking over the service when a failure occurs in themain site 101.

When the host 111 and the host 112 write data in a volume, a writingtime is attached to the written data using the host internal clock. Thetime information attached to the written data by the host is called atime stamp.

FIG. 16 is a diagram depicting an example of the functionalconfiguration or software configuration of the host 111.

The application 1601 is application software executed in the host 111 bya user, and the application reads the volume of the storage systems 131.The command-issuing program 1602 is a program whereby a routine isexecuted for issuing a pair operation command for the local replicationfunction and a pair operation command for asynchronous remote copying.The term “pair” herein refers to the pair made up of the copy sourcevolume and copy target volume of remote copying. The periodic I/Oprogram 1603 is a program whereby a routine is executed for issuingwrite data to the storage systems 131 when no I/O occurs from the host111 to the storage systems 131 for a certain period of time or longer.

FIG. 17 is a diagram depicting an example of the functionalconfiguration or software configuration of the host 112.

The application 1601 and command-issuing program 1602 of the host 112are equivalent to those of the host 111. The application 1601 of thehost 112 is usually suspended; and, when a failure occurs and service iscontinued by the sub-site, the user uses the application 1601 to resumeservice.

FIG. 2 is a diagram depicting an example of the functional configurationof the storage systems 131 contained at the main site 101.

The I/O acceptance program 201, the RC pair operation program 202, thewrite program 203, the RC data transfer program 204, the pairinformation 211, and the RC data information 212 are stored in thememory 154 of the storage systems 131. Each of the programs is executedby the processor 153 in the storage systems 131. “RC” as used herein isan abbreviation for “remote copying.” Also, the pair information 211 andthe RC data information 212 may be stored in the volume in the storagesystems.

The I/O acceptance program 201 is a program whereby a routine isexecuted for receiving the data written from the host 111.

The RC pair operation program 202 is a program whereby a routine isexecuted for performing an operation (hereinafter referred to as a pairoperation) in which a pair operation command is received from the host111, and an asynchronous remote copy pair is created.

The write program 203 is a program whereby a routine is executed forwriting, to the volume 141, the write data received by the I/Oacceptance program 201.

The RC data information 212 stores data written from the host 111 towhich information is attached for transfer by asynchronous remotecopying. The term “information to perform a transfer by asynchronousremote copying” used herein refers to the address of the logical volume(hereinafter referred to as LUID) that is the data write target, aserial number or equivalent additional information (hereinafter referredto as the SEQ#) designed to ensure sequencing, and the write time.Details thereof are shown in FIG. 5.

The RC data transfer program 204 is a program whereby a routine isexecuted for attaching information, to perform a transfer byasynchronous remote copying, to the data written in the storage systems131 from the host 111, and for transferring the data written in thestorage systems 131 from the host 111 to the storage systems 132.

The pair information 211 is information relating to the pair targetedfor asynchronous remote copying, and it is information indicating thecorrespondence of the asynchronous remote copying source logical volume141 to the asynchronous remote copying target logical volume 142, andalso indicating the state of the pair composed of the volume 141 and thevolume 142. Details thereof are shown in FIG. 4. Also, the volume 141and the volume 142 may each have a plurality of logical volumes therein.The pair information 211 shown in FIG. 4 is an example of a case inwhich a plurality of logical volumes is contained in a volume.

In asynchronous remote copying, the pair state is defined andadministered as pair information in order to indicate the copy state.The pair state of the asynchronous remote copying is information forindicating the copy state to the administrator of remote copying. Theadministrator controls the copy processing of the asynchronous remotecopying by instructing the transit of the pair state using a command.The pair state of asynchronous remote copying will be describedhereinafter.

In the present embodiment, the pair state is defined as Simplex (X),Initial-Copying (IC), Duplex (D), Suspend (S), Duplex-Pending (DP), andSuspending (SG).

The Simplex state is the state in which copying between the copy source(hereinafter referred to as a source) and the copy target (targethereinafter) has not been initiated.

The Initial-Copying state is the state in which copying is initiatedbetween the source and target volumes until transit occurs from theSimplex state to the Duplex state to be described hereinafter. Duringthe Initial-Copying state, initialization copying from the source volume(source volume hereinafter) to the target volume (target volumehereinafter), specifically, copying of the data already stored in thesource volume, is performed. When initialization copying is completedand the necessary internal processing for the transit to the Duplexstate has ended, the pair state becomes the Duplex state.

The Duplex state is the state in which initialization copying iscompleted and update copying is performed; specifically, the state inwhich the data written in the source volume is update copied in thetarget volume in a case in which data is written from the host to thesource volume. Macroscopically, the volume data is considered to be thesame between the source and target as a result of the pair statebecoming the Duplex state. However, update copying is performedasynchronously, so that the uniformity of the data stored at the mainsite and the sub-site is not strictly ensured.

The Suspend state is the state in which update copying is suspended. Inthe Suspend state, the uniformity of data between the source and targetvolumes is no longer ensured. For example, the pair state transitions tothe Suspend state upon command from an operator, the host, a computeradministrating the storage system, or the like.

When copying of data from the source volume to the target volume becomesimpossible due to a cause other than a command from an operator, thehost, a computer administrating the storage system, or the like, thestorage system automatically transitions the pair state to the Suspendstate (hereinafter referred to as the failure Suspend state). Possiblecauses for the failure Suspend state are a failure of the source volumeor target volume, a failure of the source or target storage system, acommunication channel failure between the source volume and targetvolume (in the case of the present embodiment, a failure in the network122 for connecting the storage system 101 with the storage system 102).However, another failure may also cause a failure Suspend state.

The Suspending state is the state which occurs from the Duplex stateuntil transit to the Suspend state. The failure Suspend state is alsoincluded in the Suspend state. In the present embodiment, the source andtarget storage systems perform processing for reflecting the data ofboth storage systems in the target storage system in the Suspendingstate.

The Duplex-Pending state is the state which occurs from the Suspendstate until transit to the Duplex state. In the Duplex-Pending state,the data stored in the source volume is copied to the target volume inorder to unify the data of the source volume with that of the targetvolume. After uniformity is secured between the data of the sourcevolume and that of the target volume, the pair state becomes the Duplexstate. Also, copying of the data in the Duplex-Pending state may involvea differential copy process for copying only that portion which needs tobe updated using information recorded in the update area of data writtenin the source volume or target volume during the aforementioned Suspendstate. The Initial-Copying state and Duplex-Pending state may becombined into one state and displayed on the screen of an administrationdevice, or they may be displayed in an alternating manner.

FIG. 3 is a diagram depicting an example of the functional configurationof the storage systems 132 contained at the sub-site 102.

The RC data transfer program 301, the RC pair operation program 202, theRC reflect program 302, the LR control program 303, the LR pairoperation program 304, the pair information 211, the RC data information212, the LR management information 311, and the LR pair information 312are stored in the memory 154 of the storage systems 132. Each program isexecuted by the processor 153 in the storage systems 131. “LR” as usedherein is an abbreviation for local remote copying.

The RC data transfer program 301 is a program whereby a routine isexecuted for receiving data transferred from the storage systems 131.

The RC pair operation program 202 is a program whereby a routine isexecuted for performing an operation in which a pair operation commandis received from the host 112, and an asynchronous remote copy pair iscreated.

The RC reflect program 302 is a program whereby a routine is executedfor writing, to the volume 142, the data received by the RC datatransfer program 301 in the order that the data has been written to thevolume 141 based on the SEQ# or write time. The processing executed bythe RC reflect program 302 will be referred to hereinafter as“reflecting.” Also, the method in which sequencing is ensured and avolume is written to based on the SEQ# is the same as the conventionalmethod, and so a description thereof is omitted.

In the storage systems 132, the volume 143 and the volume 144 which arereplicas of the volume 142, are created using the local replicationfunction. The local replication function is executed by both the LRcontrol program 303 and the LR pair operation program 304.

The LR control program 303 is a program whereby a routine is executedfor writing the data written in the volume 142 to the volume 143 orvolume 144 on the basis of the LR pair information 312.

The LR pair operation program 304 is a program for executing processingwhereby the following two instructions are received from the host 111 orhost 112: a Suspend command for suspending the copying to another volumeof data that is stored in a certain volume and that has a write timethat is after a certain time specified by the host 111 or host 112(hereinafter referred to as At-Time-Suspend; the details of which areillustrated in FIG. 11), and a Resync command for unifying the datastored in a certain volume with the data stored in another volume. Theroutine is performed for the state of a pair that indicates acombination of the volume 142 with the volume 143, or the volume 142with the volume 144 (hereinafter referred to as a “LR pair”). The paircomposed of the volume 142 and the volume 143 will be referred tohereinafter as LR1, and the pair composed of the volume 142 and thevolume 144 will be referred to as LR2. In At-Time-Suspend, the pairstate of the pair specified by the At-Time-Suspend command istransitioned to the Suspend state immediately before the writing of datahaving a write time that is after the time specified by theAt-Time-Suspend command.

The LR management information 311 is designed to indicate the state ofthe LR pair. Details of the LR management information 311 are shown inFIG. 6.

The LR pair information 312 is information relating to the pair of thelocal replication function and is information for indicating thecorrespondence between the copy source volume and copy target volume ofthe data, and the state of the pair. Details of the LR pair information312 are shown in FIG. 7.

In the local replication function, the pair state is defined andadministered as pair information in order to indicate the copy state.The pair state of the local replication function is information forindicating the copy state to the administrator of local replication. Theadministrator controls the copy processing of the local replicationfunction by instructing transit of the pair state using a command. Thepair state may be defined as Simplex (X), Initial-Copying (IC), Duplex(P), Suspend (S), Suspending (SG), and Resyncing (R). The pair state ofthe LR pair will be described hereinafter. States other than theResyncing state are equivalent to the definitions of the pair states ofthe RC pair described above, and so a description thereof is omitted.

The Resyncing state is the state which occurs from the Suspend stateuntil transit to the Duplex state. In the Resyncing state, copying ofdata from the source volume to the target volume is executed in order tounify the data of the source volume with that of the target volume. Whenunity is ensured between the data of the source volume and that of thetarget volume, the pair state becomes the Duplex state. Also, in thepresent embodiment, a changeover to the Suspend state occurs when thepair state is the Duplex state. Consequently, control may be performedso that the pair state becomes the Duplex state immediately after aResync command is issued, so as to promptly change the pair state fromthe Duplex state to the Suspend state. The command for instructing thepair state to become Duplex immediately after the Resync command isissued is referred to hereinafter as a Quick-Resync command. Thereexists a system whereby background copying is caused to be executedafter the pair state has become Duplex and all data copying is performedin the target volume at the time when the Quick-Resync command isissued. Another system is a system whereby all copying is performedafter the next Quick-Suspend (Quick-Suspend will be describedhereinafter) is received without performing copying in the Duplex state.

Copying of data in the Resyncing state may also be performed using adifferential copy process for copying only that portion which needs tobe updated using information recorded in the update area of data duringthe aforementioned Suspend state.

The consistency of the target volume with the data stored in the sourcevolume at the time the Suspend command was issued is ensured during theSuspend state of the local replication function.

The pair state may also become “Suspend” immediately after the Suspendcommand is issued in the local replication function of the presentembodiment. The command for instructing the pair state to become Suspendimmediately after the Suspend command is issued will be referred tohereinafter as a Quick-Suspend command. There exists a system wherebybackground copying is caused to be executed after the pair state becomesSuspend and all data copying is performed in the target volume when theQuick-Suspend command is issued, and a system whereby copying isperformed as a snapshot as needed.

FIG. 15 is a diagram depicting an example of a pair in the asynchronousremote copying and local replication function of the present embodiment.

In the pair 1501, the data stored in the volume 141 is copied to thevolume 142 in an asynchronous remote fashion. The pair 1501 is usuallyin the Duplex state.

In the pair 1502, the data stored in the volume 142 is copied to thevolume 143 by the local replication function based on the write timeattached to the data.

In the pair 1503, the data stored in the volume 142 is copied to thevolume 144 by the local replication function based on the write timeattached to the data.

In the present embodiment, the storage systems 132 are controlled sothat the pair state of either of the pair 1502 or the pair 1503 isalways in the Suspend state. The pair that is in the Duplex state is setto the Suspend state, and, after the Suspend state is completed, theother pair is set to the Resync state. By repeating this operation, datathat is consistent with the volume 142 at a certain time is stored inthe volume 143 or in the volume 144.

FIG. 4 is a diagram depicting an example of the pair information 211.

The pair information 211 has a logical volume number for the sourcevolume (hereinafter referred to as source LU ID) 401; a logical volumenumber for the target volume that corresponds to the source volume(target LU ID hereinafter) 402; a pair state 403 for indicating the pairstate of the target volume and the source volume; and a differentialbitmap 404 for indicating the memory area in which there is a furtherdifference in data between the source LU ID and the target LU ID.

FIG. 5 is a diagram depicting an example of the RC data information 212.

The RC data information 212 administers the data written in the writestorage systems 131 from the host 111; specifically, the datatransferred by asynchronous remote copying from the storage systems 131to the storage systems 132. The RC data information 212 has the targetLUID 501 of the data transferred by asynchronous remote copying, the SEQ# 502 attached by the RC data transfer program 204, the time stamp 503attached by the host 111, the data 504 transferred by asynchronousremote copying, and the Flag 505. The Flag 505 is designed to indicatethe timing at which the At-Time-Suspend command is executed. The RCreflect program 302 receives the write data for which the Flag 505 isON; specifically, for which a 1 is stored in the Flag 505, whereupon thepair state of the LR is set to the Suspend state before reflection.Details thereof are shown in FIG. 13.

FIG. 6 is a diagram depicting an example of the LR managementinformation 311.

The LR management information 311 administers the pair information ofthe LR pair and the time suspended by the At-Time-Suspend. The LRmanagement information 311 administers LR pair No. 601 for identifyingthe LR pair, the Suspend time 602 for recording the newest specifiedtime of the At-Time-Suspend for each pair, and the pair state 603 ofeach LR pair.

FIG. 7 is a diagram depicting an example of the LR pair information 312.

The LR pair information 312 administers the correspondence between thesource volume and the target volume. The LR pair information 312administers the source LU ID 701 of the LR pair and the target LU ID 702of the LR pair, the pair state 703, and the differential bitmap 704 forindicating whether a difference exists between the data of the source LUID and that of the target LU ID.

An example of the operation of the present embodiment will be describedhereinafter. In this example, an instruction from the host 111 or host112 is issued to the plurality of storage systems 131 shown in FIG. 1 orto the plurality of storage systems 132, and the plurality of storagesystems 131 or plurality of storage systems 132 execute the instructedprocessing based on the instruction from the host 111 or host 112.

FIG. 8 is a flow diagram depicting an example of the operation of the LRpairs LR1 and LR2 during normal operation. In the initial step, the LR1pair is in the Duplex state, and the LR2 pair is in the Suspend state.

First, the command-issuing program 2202 of the host 111 specifies thesame time and issues the At-Time-Suspend command of LR1 to the RCreflect program 302 of each of the plurality of storage systems 132 viathe RC data transfer program 204 of the storage systems 131 (step 801).

The LR pair operation program 304 of each of the plurality of storagesystems 132 then sets the pair state of LR1 that has assumed the Duplexstate to the Suspend state according to the At-Time-Suspend command.Details thereof are shown in FIG. 11. By setting the pair state of thepair LR1 of each of the plurality of storage systems 132 to the Suspendstate based on the same time, consistent data is stored, for the data towhich a time is attached, that is before the time specified by theAt-Time-Suspend command, in the volume 143 of the other storage systems132 at the time specified by the At-Time-Suspend command, to a certainvolume 143 of the storage systems 132 (step 802).

The command-issuing program 2202 of the host 111 then issues an LR2Resync command to the LR pair operation program 304 of each of theplurality of storage systems 132 via the storage systems 131 (step 803).

The LR pair operation program 304 of each of the plurality of storagesystems 132 then sets the pair state of LR2 that has assumed the Suspendstate to the Duplex state according to the Resync command (step 804).

The command-issuing program 2202 of the host 111 then specifies the sametime and issues the At-Time-Suspend command of LR2 to the RC reflectprogram 302 of each of the plurality of storage systems 132 via the RCdata transfer program 204 of the storage systems 131 (step 805).

The LR pair operation program 304 of each of the plurality of storagesystems 132 then sets the pair state of LR2 that has assumed the Duplexstate to the Suspend state according to the At-Time-Suspend command. Bysetting the pair state of the pair LR2 of each of the plurality ofstorage systems 132 to the Suspend state based on the same time,consistent data is stored for the data to which a time is attached thatis before the time specified by the At-Time-Suspend command in thevolume 144 of the other storage systems 132 at the time specified by theAt-Time-Suspend command to a certain volume 144 of the storage systems132 (step 806).

The command-issuing program 2202 of the host 111 then issues an LR1Resync command to the LR pair operation program 304 of each of theplurality of storage systems 132 via the storage systems 131 (step 807).

The LR pair operation program 304 then returns the pair state of LR2that has assumed the Suspend state to the Duplex state according to theResync command and returns to step 801 (step 808).

FIG. 9 is a flow diagram depicting an example of the processing whichoccurs until service is restarted in the host 112 after a failure occursat the main site.

The host 112 of the sub-site detects that a failure has occurred at themain site. Also, a failure at the main site may be detected by a systemadministrator instead of by the host 112 (step 901).

The command-issuing program 2202 of the host 112 then issues to the RCpair operation program 202 of each of the plurality of storage systems132 a command for canceling the asynchronous remote copy pair (step902).

The RC pair operation program 202 of each of the plurality of storagesystems 132 then cancels the asynchronous remote copy pair and sets thepair state to the Simplex state (step 903).

The LR control program 303 of each of the plurality of storage systems132 then elects from the volume 143 or volume 144 the newest LR volumethat is consistent with the volume 142 at a certain time (detailsthereof are shown in FIG. 12) (step 904).

The LR pair operation program 304 of each of the plurality of storagesystems 132 then sets the LR pair made up of pairs LR1 and LR2 to theSuspend state (step 905).

The LR control program 303 of each of the plurality of storage systems132 then copies, to the volume 142, the data of the LR volume elected instep 904. The LR control program 303 of each of the plurality of storagesystems 132 may instruct the LR pair operation program 304 to set the LRpair to the Resync state and to use a differential copy process whencopying the data of the LR volume elected in step 904 to the volume 142(step 906).

The LR control program 303 of each of the plurality of storage systems132 then sets, to the Suspend state, the LR pair that had been set tothe Resync state in step 906 (step 907).

The LR control program 303 of each of the plurality of storage systems132 then sets, to the Resync state, the LR pair that is not the LR pairset to the Resync state in step 905 (step 908).

Service is then restarted in the host 112 (step 909).

FIG. 10 is a flow diagram depicting an example of the processing whichoccurs until the main site 101 is recovered and service is restarted inthe host 111 after restarting service in the sub-site 102.

First, the administrator confirms recovery of the main site 101 (step1001).

The command-issuing program 2202 of the host 112 then issues aninitialization copy command for asynchronous remote copying to each ofthe plurality of storage systems 132 according to an instruction of theadministrator from an administration terminal connected to theinformation processing system (step 1002).

Each of the plurality of storage systems 132 then performsinitialization copying of asynchronous remote copying to the storagesystems 131 (step 1003).

When the administrator finishes confirming from the administrationterminal that the processing of step 1003 is completed, an instructionis issued to suspend the service of the host 112 (step 1004).

According to the command of the administrator from the administrationterminal, the RC pair operation program 202 of each of the plurality ofstorage systems 132 then changes the asynchronous remote copying, fromthe storage systems 132 to the storage systems 131, to asynchronousremote copying from the storage systems 131 to the storage systems 132(step 1005).

The administrator then confirms from the administration terminal thatthe change has been made to asynchronous remote copying from the storagesystems 131 to the storage systems 132, whereupon the command-issuingprogram 2202 of the host 112 issues a command to the LR pair operationprogram 304 of each of the plurality of storage systems 132 to set LR2to the Suspend state according to the instruction of the administratorfrom the administration terminal (step 1006).

The LR pair operation program 304 of each of the plurality of storagesystems 132 then sets LR2 to the Suspend state (step 1007).

The administrator then restarts service in the administration host 111(step 1008).

The normal operation shown in FIG. 8 is then restarted (step 1009).

FIG. 11 is a flow diagram depicting an example of the At-Time-Suspendoperation of the local replication function.

First, the command-issuing program 2202 of the host 111 issues anAt-Time-Suspend command for the LR pair to the RC data transfer program204 of each of the plurality of storage systems 131 (step 1101).

The RC data transfer program 204 of each of the plurality of storagesystems 131 then sets, to ON, the Flag 505 of the RC data information212 for the first write after the time specified by the At-Time-Suspendcommand. Also, the Flag 505 may be set to ON when the RC reflect program302 has received data instead of the RC data transfer program 204 (step1103).

The RC data transfer program 204 of each of the plurality of storagesystems 131 then transfers to the storage systems 132 the specified timeof the At-Time-Suspend command and the write data for which the Flag 505was set to the ON state (step 1104).

The LR control program 303 of the storage systems 132 then receives fromthe storage systems 131 the write data for which the Flag 505 was set tothe ON state and the specified time of the At-Time-Suspend command (step1105).

The RC reflect program 302 then issues a command to the LR pairoperation program 304 of each of the plurality of storage systems 132just before reflecting, in the volume 142, the write data for which theFlag 505 was set to ON, so as to set the LR pair specified by theAt-Time-Suspend command to the Suspend state, and rewrites the Suspendtime 602 of the LR pair information 312. The flow involved in settingthe LR pair to the Suspend state is shown in FIG. 13 (step 1106).

FIG. 12 is a flow diagram of the LR volume election operation executedby each of the plurality of storage systems 132 in step 904 of FIG. 9.

First, the LR pair operation program 304 of each of the plurality ofstorage systems 132 checks the pair state 603 of the pairs LR1 and LR2from the LR management information 311 (step 1201).

The LR pair operation program 304 of each of the plurality of storagesystems 132 then determines whether the pair state of the pairs LR1 andLR2 is the Suspend state, and the process proceeds to step 1204 if thepair state of either the LR1 or LR2 is the Suspend state, and proceedsto step 1205 if the pair state of both LR1 and LR2 is the Suspend state.At this time, if the pair state is the Suspending state or the Resyncingstate, the process waits until each is transitioned to the Suspend stateor the Resync state. No cases are encountered in which the pair state ofboth LR1 and LR2 is the Resync state or Duplex state (step 1203).

In step 1204, the LR control program 303 of each of the plurality ofstorage systems 132 elects, from among the pairs LR1 and LR2, the pairwhose pair state is the Suspend state.

In step 1205, the LR pair operation program 304 of each of the pluralityof storage systems 132 checks the Suspend time 602 of LR1 and LR2 fromthe LR management information 311 (step 1205).

The LR pair operation program 304 of each of the plurality of storagesystems 132 then elects the LR pair with the newest Suspend time 602 ofthe pairs LR1 and LR2 (step 1206).

FIG. 13 is a flow diagram depicting an example of the operation executedby each of the plurality of storage systems 132 in step 1106 of FIG. 11.

First, the RC reflect program 302 of each of the plurality of storagesystems 132 performs reflection up to the write data having the SEQ#immediately preceding the write data for which the Flag 505 is ON (step1301).

The RC reflect program 302 of each of the plurality of storage systems132 then notifies the LR pair operation program 304 that reflection iscompleted up to the point immediately preceding the write data for whichthe Flag 505 is ON (step 1302).

The LR pair operation program 304 of each of the plurality of storagesystems 132 then receives notification that reflection up to the pointimmediately preceding the write data for which the Flag 505 is ON iscompleted, whereupon the LR pair specified by the At-Time-Suspend is setto the Suspend state after the data up to the point immediatelypreceding the write data for which the Flag 505 is ON has been writtento the volume 143 or the volume 144. At this time, a Quick-Split may beused (step 1303).

The LR pair operation program 304 of each of the plurality of storagesystems 132 then notifies the RC reflect program 302 of Suspendcompletion. At this time, the pair state may be the Suspending state(step 1304).

The LR pair operation program 304 of each of the plurality of storagesystems 132 rewrites the pair state 703 of the LR pair information 312and the pair state 603 of the LR management information 311 to theSuspend state for the pair for which the pair operation was performed,and it writes the time specified by the At-Time-Suspend for the pair forwhich the pair operation was performed in regards to the Suspend time602 of the LR management information 311 (step 1305).

In At-Time-Suspend, the Suspend state is not established if writing doesnot occur after the specified time for all of the storage systems 131.In order to prevent this, when writing has not occurred for longer thana certain time for one of the storage systems 131, the periodic I/Oprogram 2203 of the host 111 carries out writing for that storagesystem. The absence of writing for longer than a certain time may alsobe prevented by initiating a write operation with the application 2201or by some other method.

In the present embodiment, the consistency, at a certain time, of datastored in the volume at a first site and the data stored in the volumeat a second site can be ensured without suspending asynchronous remotecopying from the first site to the second site in an informationprocessing system having a first site and a second site. In the specificcase of a first site having a plurality of storage systems and a secondsite having a plurality of storage systems, the consistency between thedata stored before a certain time in each of the plurality of storagesystems of the second site and the data stored before a certain time ineach of the plurality of storage systems of the first site is ensuredwhen failure occurs in the first site.

Second Embodiment

In a second embodiment, the storage system of the sub-site creates areplica of the target volume of asynchronous remote copying at a certaintime by using a snapshot function. The snapshot function provides thehost with a replica volume (hereinafter referred to as a snapshotvolume) of the copy source volume in virtual fashion. The action wherebythe storage system provides a snapshot volume to the host in virtualfashion will be referred to hereinafter as creating a snapshot volume.

The system configuration, normal operation, and recovery method afterfailure according to the second embodiment will be describedhereinafter.

The difference in the information processing system of the secondembodiment from that of the first embodiment is that storage systems1411 are used instead of the storage systems 132. Instead of the volume143 and volume 144 of the storage systems 132, the storage systems 1411have the virtual volume 1401 and the volume 1402 for storing data, andother aspects of their configuration are the same as in the storagesystems 132.

FIG. 18 is a diagram depicting an example of the functionalconfiguration of the storage systems 1411 contained at the sub-site 102.

The RC data transfer program 301, the RC pair operation program 202, theRC reflect program 302, the snapshot control program 1801, the pairinformation 211, the RC data information 212, and the snapshotinformation 1811 are stored in memory 154 in the storage systems 1411.The programs are executed by the processor 153 in the storage systems1411. Also, the pair information 211, the RC data information 212, andthe snapshot information 1811 may be stored in the volume 142 or thevolume 1402 in the storage systems 1411.

The program operation or information other than the snapshot controlprogram 1801 and the snapshot information 1811 in this arrangement isequivalent to the first embodiment.

The snapshot control program 1801 is a program whereby a routine isexecuted for controlling the data written in the volume 142 based on thesnapshot information 1811 and for controlling the creation, deletion,and the like of the snapshot volume.

The storage systems 1411 have a volume 142 for storing a copy of thedata stored in the storage systems 131, the snapshot volume 1401 inwhich snapshot data from a certain time are virtually stored, and thevolume 1402 to which the data written to the volume 142 prior to acertain time is saved if new data is written to the volume 142.

Details of the snapshot information 1811 are shown in FIG. 20.

FIG. 14 is a diagram depicting an example of a pair created byasynchronous remote copying and by the snapshot function of the presentembodiment.

In the pair 1401, the data stored in the volume 141 is copied to thevolume 142 in an asynchronous remote fashion. The pair 1401 is normallyin the Duplex state.

In the present embodiment, each of the plurality of storage systems 1411of the sub-site is controlled using the volume 1402 so as to maintain astate that always has the snapshot volume 1401. After creation of a newsnapshot volume is completed, data that is consistent at a certain timeamong the plurality of storage systems 1411 is stored by repeatingdeletion of the old snapshot volume. Details thereof will be describedhereinafter.

FIG. 20 is a diagram depicting an example of the snapshot information1811. The snapshot information 1811 has a snapshot No. 2001 and asnapshot Flag 2002.

The snapshot No. 2001 is a number for uniquely identifying the createdsnapshot volume. The snapshot Flag 2002 is designed to determine thenewest snapshot volume, and the snapshot volume for which the snapshotFlag 2002 is ON is elected during recovery of the main site from afailure.

FIG. 19 is a flow diagram depicting an example of the operation wherebya snapshot is created during normal operation.

First, the command-issuing program 2202 of the host 111 specifies thesame time and issues the At-Time-Snapshot command to the RC reflectprogram 302 of each of the plurality of storage systems 1411 via the RCdata transfer program 204 of the storage systems 131 (step 1901). TheAt-Time-Snapshot command is a command for executing the snapshotfunction.

The snapshot control program 1801 of each of the plurality of storagesystems 1411 then creates a snapshot volume according to the specifiedtime of the At-Time-Snapshot command. Details thereof are shown in FIG.23 (step 1902).

The snapshot control program 1801 of each of the plurality of storagesystems 1411 then notifies the host 111 that snapshot creation iscompleted when the snapshot volume is created (step 1903).

When notification of completion of snapshot creation is received fromall of the snapshot control programs 1801 of each of the plurality ofstorage systems 1411, the command-issuing program 2202 of the host 111issues a command to delete the old snapshot volume to the snapshotcontrol program 1801 of each of the plurality of storage systems 1411via the RC data transfer program 204 of the storage systems 131. At thistime, the host 111 issues an instruction so as to set the snapshot Flag2002 of the snapshot volume created in step 1902 to ON and the snapshotFlag 2002 of the other snapshot volume to OFF in the storage systems1411 (step 1904).

The snapshot control program 1801 of each of the plurality of storagesystems 1411 then deletes the old snapshot volume, and the processreturns to step 1901 (step 1905). Deletion of the old snapshot volumeherein refers to the snapshot control program 1801 deleting the datastored in the saving volume 1402 that corresponds to the snapshot volumespecified by the delete command, and deleting the snapshot No. 2001 ofthe old snapshot volume. After the old snapshot volume is deleted, thesnapshot control program 1801 notifies the host of the deletion of theold snapshot volume. The host 111 that has received notification thatdeletion of the old snapshot volume is completed changes the setting inthe host to the old snapshot volume. Also, the snapshot control programreturns an error to the host if the snapshot control program 1801 hasdeleted the old snapshot volume, and the host has accessed the oldsnapshot volume. The host 111 may then change the setting in the host tothe old snapshot volume.

FIG. 23 is a flow diagram depicting an example of the At-Time-Snapshotoperation of the snapshot function.

First, the command-issuing program 2202 of the host 111 issues theAt-Time-Snapshot command to the RC data transfer program 204 of each ofthe plurality of storage systems 131 (step 2301).

The RC data transfer program 204 of each of the plurality of storagesystems 131 then sets the Flag 505 of the RC data information 212 to ONfor the first write after the time specified by the At-Time-Snapshotcommand. Also, the RC reflect program 302 may set the Flag 505 to ONwhen it has received the data, instead of the RC data transfer program204 (step 2302).

The RC data transfer program 204 of each of the plurality of storagesystems 131 then transfers, to the storage systems 1411, the specifiedtime of the At-Time-Snapshot command and the write data for which theFlag 505 is ON (step 2303).

The LR control program 303 of the storage systems 1411 then receives,from the storage systems 131, the write data for which the Flag 505 isON and the specified time of the At-Time-Snapshot (step 2304).

With the RC reflect program 302 of each of the plurality of storagesystems 1411, the data up to that immediately preceding the writing forwhich the Flag 505 is ON is then reflected to the volume 142 (step2305).

The RC reflect program 302 of each of the plurality of storage systems1411 then notifies each snapshot control program 1801 that reflection iscompleted up to the data immediately prior to writing for which the Flag505 is ON (step 2306).

The snapshot control program 1801 then creates the snapshot volume 1401(step 2307).

The snapshot control program 1801 of each of the plurality of storagesystems 1411 then notifies each RC reflect program 302 of snapshotcreation (step 2308).

The snapshot control program 1801 of each of the plurality of storagesystems 1411 then writes the number of the snapshot created earlier inthe snapshot information 1811 (step 2309). Step 2309 may be performedbefore step 2308.

When the snapshot volume is created and data is transferred from thestorage systems 131 to the storage systems 1411, the data, which isstored in the memory area that holds the data transferred from thestorage systems 131 to the storage systems 1411, is stored in the volume1402. The data transferred from the storage systems 131 to the storagesystems 1411 is then stored in the volume 142. However, this processingis executed only when the write operation is the first write operationto the memory area that occurs after the newest snapshot volume iscreated. Specifically, whether the writing is the first is managed bythe bitmap in the storage systems 1411, and this processing is executedonly in the case of the first write operation. The bitmap records thesnapshot No. 2001, the address in the saving volume 1402 containing thesaved data stored in the memory area of the volume 141 before the firstwrite was executed, and the result of determining the address of thesnapshot volume to which these data correspond. Also, whether the writeoperation is the first write may be administered by a method other thanadministration by a bitmap.

FIG. 21 is a flow diagram depicting an example of the processing whichoccurs until restarting of service by the host 112 after a failureoccurs in the main site.

The host 112 of the sub-site issues notification that a failure hasoccurred in the main site. Also, notification of failure in the mainsite may be issued by the system administrator instead of by the host112 (step 2101).

The command-issuing program 1602 of the host 112 then provides the RCpair operation program 202 of each of the plurality of storage systems1411 with a command to cancel the asynchronous remote copy pair (step2102).

The RC pair operation program 202 of each of the plurality of storagesystems 1411 then cancels the asynchronous remote copy pair and sets thepair state to the Simplex state (step 2103).

The snapshot control program 1801 of each of the plurality of storagesystems 1411 then elects the snapshot volume for which the snapshot Flag2002 is ON (step 2104).

The snapshot control program 1801 of each of the plurality of storagesystems 1411 then copies, to the volume 142, the data of the snapshotvolume elected in step 2104. Specifically, the bitmap is referenced andthe data stored in the volume 1402, that corresponds to the snapshotvolume for which the snapshot Flag 2002 is ON is specified for copyingto the volume 142. Also, in the present embodiment, the volume 1402 wasshared by all of the snapshot volumes to which a snapshot number wasattached, but a volume 1402 may also be prepared for each snapshotvolume. In this case, the data stored in the volume 1402 thatcorresponds to the elected snapshot volume is copied to the volume 142(step 2105).

The service is then restarted by the host 112 (step 2106).

FIG. 22 is a flow diagram depicting an example of the processing whichoccurs until the main site 101 is recovered and service is restarted bythe host 111 after service is restarted at the sub-site 102.

First, the administrator confirms recovery of the main site 101 (step2201).

The command-issuing program 2202 of the host 112 then issues aninitialization copy command for asynchronous remote copying to each ofthe plurality of storage systems 132 according to an instruction of theadministrator from an administration terminal connected to theinformation processing system (step 2202).

Each of the plurality of storage systems 1411 then performsinitialization copying of asynchronous remote copying to the storagesystems 131 (step 2203).

When the administrator finishes confirming, from the administrationterminal, that the processing of step 1003 is completed, an instructionis issued to suspend the service of the host 112 (step 2204).

According to the command of the administrator from the administrationterminal, the RC pair operation program 202 of each of the plurality ofstorage systems 132 then changes the asynchronous remote copying fromthe storage systems 1411 to the storage systems 131 to asynchronousremote copying from the storage systems 131 to the storage systems 1411(step 2205).

The command-issuing program 1602 of the host 112 then issues a commandinstructing snapshot creation to each of the plurality of storagesystems 1411 (step 2206).

The plurality of storage systems 1411 then create a snapshot (step2207).

The administrator then restarts service in the host 111 (step 2208).

The normal operation shown in FIG. 19 is then restarted (step 2209).

In the present embodiment, only the difference in the data between thevolume 142 and the snapshot volume 1401 is saved in the volume 1402, andso it becomes possible to reduce the volume capacity and the amount ofdata transferred in comparison to a case in which the local replicationfunction is used.

1. An information processing system comprising: a first site that isconnected to a first computer and has a first storage system and asecond storage system; and a second site having a third storage systemconnected to a second computer and to the first storage system, and afourth storage system connected to the second computer and to the secondstorage system; wherein the third storage system has a first memory areafor storing data transferred from the first storage system, and a secondmemory area and third memory area for storing a copy of the data storedin the first memory area of the third storage system; the fourth storagesystem has a first memory area for storing data transferred from thefirst storage system, and a second memory area and third memory area forstoring a copy of the data stored in the first memory area of the fourthstorage system; the first storage system and the second storage systemeach receive data with an attached writing time from the first computer;the first storage system transfers to the third storage system the datareceived from the first computer and a copy of the writing time; thesecond storage system transfers to the fourth storage system the datareceived from the first computer and a copy of the writing time; thethird storage system writes to the first memory area in the thirdstorage system the data received from the first storage system in thetime sequence in which the data were attached, and writes to the secondmemory area in the third storage system a copy of the data written inthe first memory area in the third storage system in the time sequencein which the data were attached; the fourth storage system writes to thefirst memory area in the fourth storage system the data received fromthe second storage system in the time sequence in which the data wereattached, and writes to the second memory area in the fourth storagesystem a copy of the data written in the first memory area in the fourthstorage system in the time sequence in which the data were attached;when copying is completed to the second memory area in the third storagesystem of the data with an attached writing time that is prior to afirst prescribed time specified by the first computer, the third storagesystem suspends writing to the second memory area in the third storagesystem of a copy of the data written in the first memory area in thethird storage system; the third storage system then initiates writing ofa copy of the data written in the first memory area in the third storagesystem to the third memory area in the third storage system; whencopying is completed to the second memory area in the fourth storagesystem of the data with an attached writing time that is prior to afirst prescribed time specified by the first computer, the fourthstorage system suspends writing to the second memory area in the fourthstorage system of a copy of the data written in the first memory area inthe fourth storage system; and the fourth storage system then initiateswriting of a copy of the data written in the first memory area in thefourth storage system to the third memory area in the fourth storagesystem.
 2. The information processing system according to claim 1,wherein: when copying is completed to the first memory area in the thirdstorage system of the data with an attached writing time that is priorto a second prescribed time specified by the first computer, the thirdstorage system suspends writing to the third memory area in the thirdstorage system; the third storage system then copies to the secondmemory area in the third storage system the data stored in the firstmemory area in the third storage system; when copying is completed tothe first memory area in the fourth storage system of the data with anattached writing time that is prior to a second prescribed timespecified by the first computer, the fourth storage system suspendswriting to the third memory area in the fourth storage system; and thefourth storage system then copies to the second memory area in thefourth storage system the data stored in the first memory area in thefourth storage system.
 3. The information processing system according toclaim 2, wherein: when a failure occurs in the first site, the thirdstorage system elects the memory area in which there was a suspension ofwriting of a copy of the data stored in the first memory area in thethird storage system among the second memory area and third memory areain the third storage system when the failure occurred, and copies thedata stored in the elected memory area to the first memory area; and thefourth storage system elects the memory area in which there was asuspension of writing of a copy of the data stored in the first memoryarea in the fourth storage system among the second memory area and thirdmemory area in the fourth storage system when the failure occurred, andcopies the data stored in the elected memory area to the first memoryarea.
 4. The information processing system according to claim 3,wherein: when the first site has recovered, the third storage systemtransfers a copy of the data stored in the first memory area in thethird storage system to the first storage system; and the fourth storagesystem transfers a copy of the data stored in the first memory area inthe fourth storage system to the first storage system.
 5. A copy methodfor an information processing system that comprises: a first site thatis connected to a first computer and has a first storage system and asecond storage system; and a second site having a third storage systemthat is connected to a second computer and to the first storage systemand that comprises a first memory area for storing data transferred fromthe first storage system, and a second memory area and third memory areafor storing a copy of the data stored in the first memory area of thethird storage system; and a fourth storage system that is connected tothe second computer and to the second storage system and that comprisesa first memory area for storing data transferred from the first storagesystem, and a second memory area and third memory area for storing acopy of the data stored in the first memory area of the fourth storagesystem; said copy method for an information processing systemcomprising: a step in which the first storage system and the secondstorage system each receive data with an attached writing time from thefirst computer; a step in which the data received from the firstcomputer and a copy of the writing time are transferred from the firststorage system to the third storage system; a step in which the datareceived from the first computer and a copy of the writing time aretransferred from the second storage system to the fourth storage system;a step in which the data received from the first storage system arewritten to the first memory area in the third storage system in the timesequence in which the data were attached, and a copy of the data writtenin the first memory area in the third storage system is written to thesecond memory area in the third storage system in the time sequence inwhich the data were attached; a step in which the data received from thesecond storage system are written to the first memory area in the fourthstorage system in the time sequence in which the data were attached, anda copy of the data written in the first memory area in the fourthstorage system is written to the second memory area in the fourthstorage system in the time sequence in which the data were attached; astep in which writing to the second memory area in the third storagesystem of a copy of the data written in the first memory area in thethird storage system is suspended when copying is completed to thesecond memory area in the third storage system of the data with anattached writing time that is prior to a first prescribed time specifiedby the first computer; a step in which writing of a copy of the datawritten in the first memory area in the third storage system to thethird memory area in the third storage system is then initiated; a stepin which writing to the second memory area in the fourth storage systemof a copy of the data written in the first memory area in the fourthstorage system is suspended when copying is completed to the secondmemory area in the fourth storage system of the data with an attachedwriting time that is prior to a first prescribed time specified by thefirst computer; and a step in which writing of a copy of the datawritten in the first memory area in the fourth storage system to thethird memory area in the fourth storage system is then initiated.
 6. Thecopy method for an information processing system according to claim 5,comprising: a step in which writing to the third memory area in thethird storage system is suspended when copying is completed to the firstmemory area in the third storage system of the data with an attachedwriting time that is prior to a second prescribed time specified by thefirst computer; a step in which the data stored in the first memory areain the third storage system are copied to the second memory area in thethird storage system; a step in which writing to the third memory areain the fourth storage system is suspended when copying is completed tothe first memory area in the fourth storage system of the data with anattached writing time that is prior to a second prescribed timespecified by the first computer; and a step in which the data stored inthe first memory area in the fourth storage system are copied to thesecond memory area in the fourth storage system.
 7. The copy method foran information processing system according to claim 6, comprising: astep in which a failure in the first site causes the memory area inwhich there was a suspension of writing of a copy of the data stored inthe first memory area in the third storage system to be elected amongthe second memory area and third memory area in the third storage systemwhen the failure occurs; a step in which the data stored in the electedmemory area are copied to the first memory area in the third storagesystem; a step in which the memory area in which there was a suspensionof writing of a copy of the data stored in the first memory area in thefourth storage system is elected among the second memory area and thirdmemory area in the fourth storage system when a failure occurs; and astep in which the data stored in the elected memory area are copied tothe first memory area in the fourth storage system.
 8. The copy methodfor an information processing system according to claim 7, comprising: astep in which a copy of the data stored in the first memory area in thethird storage system is transferred to the first storage system when thefirst site has recovered; and a step in which a copy of the data storedin the first memory area in the fourth storage system is transferred tothe second storage system.
 9. An information processing system,comprising a first site that is connected to a first computer and has afirst storage system and a second storage system; and a second sitehaving a third storage system connected to a second computer and to thefirst storage system, and a fourth storage system connected to thesecond computer and to the second storage system; wherein the thirdstorage system has a first memory area for storing data transferred fromthe first storage system, and a second memory area which is anevacuation destination of the data stored in the first memory area ofthe third storage system; the fourth storage system has a first memoryarea for storing data transferred from the second storage system, and asecond memory area which is an evacuation destination of the data storedin the first memory area of the fourth storage system; the first storagesystem and the second storage system each receive data with an attachedwriting time from the first computer; the first storage system transfersto the third storage system the data received from the first computerand a copy of the writing time; the second storage system transfers tothe fourth storage system the data received from the first computer anda copy of the writing time; the third storage system receives data withan attached writing time that is subsequent to the first prescribed timespecified by the first computer, whereupon if the data stored in thefirst memory area of the third storage system in which data are writtenwith a writing time attached thereto that is subsequent to the firstprescribed time are data that precede the first prescribed time, thedata are copied to the second memory area of the third storage system,and data to which a writing time was attached that is subsequent to thefirst prescribed time are stored in the first memory area in the thirdstorage system; the data stored in the second memory area in the thirdstorage system are deleted when data are received to which a writingtime is attached that is subsequent to the second prescribed timespecified by the first computer; the fourth storage system receives datawith an attached writing time that is subsequent to the first prescribedtime, whereupon if the data stored in the first memory area of thefourth storage system in which data are written with a writing timeattached thereto that is subsequent to the first prescribed time aredata that precede the first prescribed time, the data are copied to thesecond memory area of the fourth storage system, and data to which awriting time was attached that is subsequent to the first prescribedtime are stored in the first memory area in the fourth storage system;and the data with an attached writing time that is prior to a firstprescribed time stored in the second memory area in the fourth storagesystem are deleted when data are received to which a writing time isattached that is subsequent to the second prescribed time.
 10. Theinformation processing system according to claim 9, wherein the thirdstorage system stores in the first memory area of the third storagesystem the data with an attached writing time that is subsequent to thesecond prescribed time after copying the data to the second memory areaof the third storage system if the data stored in the first memory areaof the third storage system containing written data with an attachedwriting time that is subsequent to the second prescribed time are datathat precede the second prescribed time; and the fourth storage systemstores in the first memory area of the fourth storage system the datawith an attached writing time that is subsequent to the secondprescribed time after copying the data to the second memory area of thefourth storage system if the data stored in the first memory area of thefourth storage system containing written data with an attached writingtime that is subsequent to the second prescribed time are data thatprecede the second prescribed time.
 11. The information processingsystem according to claim 9, wherein: when a failure occurs in the firstsite, the third storage system copies data having a writing time thatprecedes the second prescribed time from the second memory area to thefirst memory area in the third storage system; and the fourth storagesystem copies data having a writing time that precedes the secondprescribed time from the second memory area to the first memory area inthe fourth storage system.
 12. The information processing systemaccording to claim 11, wherein when the first site has recovered, thethird storage system transfers a copy of the data stored in the firstmemory area in the third storage system to the first storage system; andthe fourth storage system transfers a copy of the data stored in thefirst memory area in the fourth storage system to the first storagesystem.